ci: add canary-rollout-tests.yml — enforce the engine's bats suite (CI gap from #613/#624)#685
Conversation
…s CI gap from #613/#624 relocation) The canary-rollout engine + tests/canary_rollout.bats were relocated into .github by #613/#624 but no CI runner came with them, so the 149+ test suite has been unenforced ever since — every #668 increment (#672/#676/#678/#684) merged without its tests actually running in CI. This workflow runs shellcheck + the full bats suite on any PR touching the engine, its pure core, the suite, or this workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J4z5uYVjwdyQjh2wFZxkut
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
|
Warning Review limit reached
Next review available in: 56 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a GitHub Actions workflow that runs ShellCheck and the canary-rollout Bats suite for relevant pull requests and pushes to ChangesCanary rollout CI
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (applied)Changes committed and pushed. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/canary-rollout-tests.yml:
- Around line 52-55: Update the actions/checkout step in the canary rollout
workflow to set persist-credentials to false, while preserving the existing
fetch-depth and checkout behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 2fd71484-07f4-4d77-bc4b-e71769b87402
📒 Files selected for processing (1)
.github/workflows/canary-rollout-tests.yml
|
@coderabbitai resolve |
Dev-Lead — review-changes (applied)Changes committed and pushed. |
✅ Action performedComments resolved and changes approved. |
|
Dev-Lead — waiting on PR blockers (intent: fix-reviews)PR: #685 |
donpetry-bot
left a comment
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 34383519be8fb02915c07f82c897a4f62fc186e3
Review mode: triage-approved (single reviewer)
Summary
Adds a single new CI workflow (.github/workflows/canary-rollout-tests.yml, +74/-0) that closes a real gap: the canary-rollout engine and its bats suite were relocated in #613/#624 without a CI runner, so the suite has been unenforced. The workflow follows the house *-tests.yml pattern (shellcheck -x + bats), is path-filtered to the engine/core/suite/workflow files, uses read-only permissions, per-ref cancellable concurrency, and a SHA-pinned checkout. The workflow ran on this PR itself and passed (Lint and bats: SUCCESS), confirming the suite is green.
Linked issue analysis
No linked issues (closingIssuesReferences is empty). The PR body clearly documents the motivating gap (#613/#624 relocation without CI) and the follow-up plan (#684 rebase, #668 increments 5-6). Self-verifying by design: this PR's own CI run is the proof the suite passes.
Findings
No blocking findings.
- Security posture is clean: permissions restricted to contents: read; actions/checkout pinned to 9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0, verified via GitHub API to match the v7.0.0 tag (per AGENTS.md pin policy); persist-credentials: false set; no pull_request_target; no untrusted ${{ }} interpolation in run blocks; GH_TOKEN is an inert dummy string for stubbed tests, not a credential.
- Prior CodeRabbit changes-requested review (persist-credentials) was addressed at head SHA; the single review thread is resolved and CodeRabbit approved the head commit.
- Secret scan: run_secret_scanning MCP tool unavailable in this environment; gitleaks CI check passed (SUCCESS) and manual inspection of the diff shows no secrets.
- Minor, non-blocking: the path filters reference tests/canary_rollout.bats and the two scripts, which exist per the PR context; if the engine files are ever renamed the filters must move with them.
CI status
All quality gates green: Lint, ShellCheck, Agent Security Scan, Secret scan (gitleaks), CodeQL (Analyze actions), SonarCloud (quality gate passed, 0 new issues/hotspots), Dependency audit, CodeRabbit, and the new Canary-rollout Tests job itself (SUCCESS). Three dev-lead dispatch/ci-relay runs show CANCELLED — these are org agent-orchestration runs superseded by concurrency on rapid successive commits, not code-quality checks. Dependabot/pip/cargo/govulncheck/pnpm audits SKIPPED (not applicable ecosystems). mergeStateStatus BLOCKED reflects the pending org-leads review requirement this review satisfies.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fa2e467f6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| paths: | ||
| - 'scripts/canary-rollout.sh' | ||
| - 'scripts/lib/canary-rollout.sh' | ||
| - 'tests/canary_rollout.bats' | ||
| - '.github/workflows/canary-rollout-tests.yml' |
There was a problem hiding this comment.
This paths list leaves out standards/canary-rings.json, even though tests/canary_rollout.bats reads that file as RINGS and contains dedicated assertions for its ring/gate shape. A PR that changes only the registry or gate knobs will therefore skip the workflow entirely, letting broken rollout metadata merge without the bats suite this workflow is meant to enforce; add the registry file to both the pull_request and push path filters.
Useful? React with 👍 / 👎.
Review — fix requested (cycle 1/3)The automated review identified the following issues. Please address each one: Findings to fixAutomated review — NEEDS HUMAN REVIEWRisk: MEDIUM SummaryAdds a single new workflow (.github/workflows/canary-rollout-tests.yml, +74/-0) that closes a real CI gap: the canary-rollout engine and its bats suite were relocated by #613/#624 without a runner. The workflow itself is well-built — action pinned by SHA (actions/checkout v7.0.0), Linked issue analysisNo linked closing issues. The PR body substantively documents the motivation (unenforced test suite since #613/#624) and gates follow-ups (#684 rebase, #668 increments 5–6) on this landing. Context is sound. Findings1. Unresolved review thread with a verified-valid finding (blocking auto-approval). chatgpt-codex-connector (P2) flagged that the path filters omit 2. Secret scan: 3. Prior reviews: CodeRabbit's earlier CHANGES_REQUESTED (persist-credentials) was addressed and it re-approved. A prior pr-review-agent approval exists at this same SHA, but the codex thread (created 21:17Z) post-dates it and remains open. CI statusAll green at head 3438351: Lint and bats (this new workflow, first run) ✓, ShellCheck ✓, CodeQL ✓, Secret scan (gitleaks) ✓, SonarCloud quality gate ✓, AgentShield ✓, CodeRabbit ✓, npm audit ✓. Cancelled dev-lead dispatch/ci-relay entries are concurrency supersessions; the latest runs succeeded. Language-specific audits skipped (no matching ecosystems). Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review. Additional tasks
The review cascade will automatically re-review after new commits are pushed. |
2 participants
Why
The canary-rollout engine (
scripts/canary-rollout.sh,scripts/lib/canary-rollout.sh) and its 149+ testtests/canary_rollout.batswere relocated into.githubby #613/#624, but no CI runner came with them. Result: the suite has been unenforced — every #668 increment (#672/#676/#678, and #684 in flight) merged without its bats tests running in CI. Discovered while shepherding #684.What
A
Canary-rollout Testsworkflow mirroring the house*-tests.ymlpattern (seeauto-rebase-tests.yml): on any PR/push touching the engine, pure core, suite, or this workflow, it runsshellcheck -x+bats --print-output-on-failure tests/canary_rollout.bats.The point
This PR's own CI run is the verification — it will show, for the first time in Actions, whether the suite is green. If red, we've surfaced real breakage (or stale test stubs) that the missing runner had been hiding.
Follow-ups gated on this:
Summary by CodeRabbit